A Primer on Motion Capture with Deep Learning: Principles, Pitfalls and Perspectives

Extracting behavioral measurements non-invasively from video is stymied by the fact that it is a hard computational problem. Recent advances in deep learning have tremendously advanced predicting posture from videos directly, which quickly impacted neuroscience and biology more broadly. In this primer we review the budding field of motion capture with deep learning. In particular, we will discuss the principles of those novel algorithms, highlight their potential as well as pitfalls for experimentalists, and provide a glimpse into the future.Comment: Review, 21 pages, 8 figures and 5 boxe

AmadeusGPT: a natural language interface for interactive animal behavioral analysis

The process of quantifying and analyzing animal behavior involves translating the naturally occurring descriptive language of their actions into machine-readable code. Yet, codifying behavior analysis is often challenging without deep understanding of animal behavior and technical machine learning knowledge. To limit this gap, we introduce AmadeusGPT: a natural language interface that turns natural language descriptions of behaviors into machine-executable code. Large-language models (LLMs) such as GPT3.5 and GPT4 allow for interactive language-based queries that are potentially well suited for making interactive behavior analysis. However, the comprehension capability of these LLMs is limited by the context window size, which prevents it from remembering distant conversations. To overcome the context window limitation, we implement a novel dual-memory mechanism to allow communication between short-term and long-term memory using symbols as context pointers for retrieval and saving. Concretely, users directly use language-based definitions of behavior and our augmented GPT develops code based on the core AmadeusGPT API, which contains machine learning, computer vision, spatio-temporal reasoning, and visualization modules. Users then can interactively refine results, and seamlessly add new behavioral modules as needed. We benchmark AmadeusGPT and show we can produce state-of-the-art performance on the MABE 2022 behavior challenge tasks. Note, an end-user would not need to write any code to achieve this. Thus, collectively AmadeusGPT presents a novel way to merge deep biological knowledge, large-language models, and core computer vision modules into a more naturally intelligent system. Code and demos can be found at: https://github.com/AdaptiveMotorControlLab/AmadeusGPT.Comment: demo available https://github.com/AdaptiveMotorControlLab/AmadeusGP

AcinoSet: A 3D Pose Estimation Dataset and Baseline Models for Cheetahs in the Wild

Animals are capable of extreme agility, yet understanding their complex dynamics, which have ecological, biomechanical and evolutionary implications, remains challenging. Being able to study this incredible agility will be critical for the development of next-generation autonomous legged robots. In particular, the cheetah (acinonyx jubatus) is supremely fast and maneuverable, yet quantifying its whole-body 3D kinematic data during locomotion in the wild remains a challenge, even with new deep learning-based methods. In this work we present an extensive dataset of free-running cheetahs in the wild, called AcinoSet, that contains 119,490 frames of multi-view synchronized high-speed video footage, camera calibration files and 7,588 human-annotated frames. We utilize markerless animal pose estimation to provide 2D keypoints. Then, we use three methods that serve as strong baselines for 3D pose estimation tool development: traditional sparse bundle adjustment, an Extended Kalman Filter, and a trajectory optimization-based method we call Full Trajectory Estimation. The resulting 3D trajectories, human-checked 3D ground truth, and an interactive tool to inspect the data is also provided. We believe this dataset will be useful for a diverse range of fields such as ecology, neuroscience, robotics, biomechanics as well as computer vision.Comment: Code and data can be found at: https://github.com/African-Robotics-Unit/AcinoSe

A land-to-ocean perspective on the magnitude, source and implication of DIC flux from major Arctic rivers to the Arctic Ocean

Author Posting. © American Geophysical Union, 2012. This article is posted here by permission of American Geophysical Union for personal use, not for redistribution. The definitive version was published in Global Biogeochemical Cycles 26 (2012): GB4018, doi:10.1029/2011GB004192.A series of seasonally distributed measurements from the six largest Arctic rivers (the Ob', Yenisey, Lena, Kolyma, Yukon and Mackenzie) was used to examine the magnitude and significance of Arctic riverine DIC flux to larger scale C dynamics within the Arctic system. DIC concentration showed considerable, and synchronous, seasonal variation across these six large Arctic rivers, which have an estimated combined annual DIC flux of 30 Tg C yr−1. By examining the relationship between DIC flux and landscape variables known to regulate riverine DIC, we extrapolate to a DIC flux of 57 ± 9.9 Tg C yr−1for the full pan-arctic basin, and show that DIC export increases with runoff, the extent of carbonate rocks and glacial coverage, but decreases with permafrost extent. This pan-arctic riverine DIC estimate represents 13–15% of the total global DIC flux. The annual flux of selected ions (HCO3−, Na+, Ca2+, Mg2+, Sr2+, and Cl−) from the six largest Arctic rivers confirms that chemical weathering is dominated by inputs from carbonate rocks in the North American watersheds, but points to a more important role for silicate rocks in Siberian watersheds. In the coastal ocean, river water-induced decreases in aragonite saturation (i.e., an ocean acidification effect) appears to be much more pronounced in Siberia than in the North American Arctic, and stronger in the winter and spring than in the late summer. Accounting for seasonal variation in the flux of DIC and other major ions gives a much clearer understanding of the importance of riverine DIC within the broader pan-arctic C cycle.Funding for this work was provided through NSF-OPP-0229302 and NSF-OPP-0732985. Additional support to SET was provided by an NSERC Postdoctoral Fellowship.2013-06-1

Motor control: Neural correlates of optimal feedback control theory

Recent work is revealing neural correlates of a leading theory of motor control. By linking an elegant series of behavioral experiments with neural inactivation in macaques with computational models, a new study shows that premotor and parietal areas can be mapped onto a model for optimal feedback control

Pretraining boosts out-of-domain robustness for pose estimation

Deep neural networks are highly effective tools for human and animal pose estimation. However, robustness to out-of-domain data remains a challenge. Here, we probe the transfer and generalization ability for pose estimation with two architecture classes (MobileNetV2s and ResNets) pretrained on ImageNet. We generated a novel dataset of 30 horses that allowed for both within-domain and out-of-domain (unseen horse) testing. We find that pretraining on ImageNet strongly improves out-of-domain performance. Moreover, we show that for both pretrained and networks trained from scratch, better ImageNet-performing architectures perform better for pose estimation, with a substantial improvement on out-of-domain data when pretrained. Collectively, our results demonstrate that transfer learning is particularly beneficial for out-of-domain robustness